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Abstract 

Measuring news media literacy is important in order for it to thrive in a variety of educational and civic 
contexts. This research builds on existing measures of news media literacy and two new scales are presented 
that measure self-perceived media literacy (SPML) and perceptions of the value of media literacy (VML). 
Research with a larger sample of college undergraduate students and a smaller sample of adults enabled the 
validation of these measures. Results confirm the value of conceptualizing news media literacy using the 
theoretical subcomponents of authors & audiences, messages & meaning, and representation & realities. The 
VML scale, in particular, proved especially consequential in predicting knowledge about and attitudes towards 
the media. 
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New media technologies are reshaping how we think about the role of the news media in democratic 
societies. Young people today learn about news and current events from an increasingly diverse set of sources, 
which can include traditional and emerging news outlets online as well as social media websites (Mitchell & 
Page 2015; Shearer 2015). As such, news media literacy is increasingly important to help individuals learn the 
skills to critically engage with news and exert control over their relationship with mediated messages (Hobbs 
2010a; Kahne, Lee, & Feezell 2012; Potter 2004). News media literacy should promote knowledge of media 
structures and a skeptical attitude towards news content (Ashley, Poepsel, & Willis 2010; Maksl, Ashley, & 
Craft 2015; Mihailidis 2009). 

To date, there has been considerable divergence in how scholars define, implement, and measure news 
media literacy (Ashley, Maksl, & Craft 2013; Hobbs, 2011; Maksl et al., 2015; Potter, 2010). While some 
conceptualize media literacy as a skillset used to analyze, evaluate, and communicate messages (Hobbs & Frost 
2003; Livingstone 2004), others focus upon the economic structures of news media ownership and control (Chu 
& Lee 2014; Duran, Yousman, Walsh, & Longshore 2008; Fleming 2014). Yet others argue that a critical 
orientation towards the news media and skepticism towards its products is an important outcome of news media 
literacy (Ashley et al. 2010; Maksl et al., 2015; Potter, 2004). Generally, scholars agree that effective media 
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literacy education should prepare people to become critical thinkers who can read, analyze, and deconstruct 
media messages and should empower individuals to understand, develop, and share infonned viewpoints using 
tools available in a digital media environment (Ashley et al. 2013; Maksl et al. 2015; Mihailidis 2014). 

As a construct, news media literacy is difficult to measure because the literature lacks both a shared 
definition and an agreed-upon method for measuring this multi-dimensional construct (Ashley et al. 2013; 

Maksl et al. 2015). The goals of this study are to validate existing news media literacy measures developed by 
Ashley et al. (2013) and to test additional news media literacy measures that focus on individuals’ self- 
perceived media literacy and societal value of media literacy (Tully & Vraga in press; Vraga & Tully 2015). We 
test these measures across two studies: a sample of college undergraduates, often the target of media literacy 
education, and a more diverse sample of adults using Amazon’s Mechanical Turk. Across both samples, we 
demonstrate that news media literacy is a multi-dimensional construct, and measurement choices impact the 
relationships with knowledge and media attitudes. 

News Media Literacy 

News has a unique role in democratic societies as it is expected to inform citizens to enable them to 
make sound democratic decisions. Therefore, it is important to interrogate news and its societal implications 
(Fleming 2014; Hobbs 2010b) News media literacy education is designed to teach individuals how to apply core 
media literacy skills (i.e., analyzing and evaluating content) to news (Ashley et al. 2013; Maksl et al. 2015; 
Mihailidis 2014; Vraga & Tully 2015). News media literacy education typically emphasizes three related 
aspects of news: (1) the conditions and constraints under which news is produced, (2) the goal of journalism to 
create an infonned public capable of making democratic decisions, and (3) the responsibility of audiences to be 
critical thinkers when consuming news content (Ashley et al. 2013; Fleming 2014; Maksl et al. 2015; Vraga 
Tully, & Rojas 2009; Vraga & Tully 2015). In short, audiences can be better prepared to interrogate news 
content if “they have a more complete understanding of the conditions in which news is produced” (Ashley et 
al. 2013, 7). Hobbs (2010b) notes that news literacy courses should focus on developing critical thinking and 
communication skills to create more informed students capable of navigating our complex news world. 

Measuring news media literacy presents an opportunity to assess media literacy curricula and to 
understand baseline levels of news media literacy among Americans. Building on previous media literacy 
research (Potter 2004; Primack et al. 2006), Ashley et al. (2013) developed a scale to measure news media 
literacy with a focus on news production techniques and consumption. Basing their model on Primack et al.’s 
(2006) “smoking media literacy scale,” Ashley et al.’s (2013) model included items to measure authors and 
audiences (AA) focusing on how authors target audiences; messages and meanings (MM) addressing the values 
and production techniques in messages that appeal to different viewers; and representation and reality (RR) 
focusing on the filtering of infonnation in the media and how this affects perceptions of reality. The final scale 
included 15 items representing all three conceptual areas (see Table 1). Our first goal in this study is testing and 
validating this News Media Literacy (NML) scale among diverse audiences - and specifically testing whether a 
single-factor model of NML is more appropriate than examining each of the three components separately. 
Despite developing their NML scale along three discrete conceptual areas, Ashley et al. (2013) argue that a 
single factor of NML emerges, an argument that we will test in this study. 

We focus on the validation of the Ashley et al. (2013) scale, rather than other potential measures of news 
media literacy (e.g., Maskl et al. 2015), given its rigorous theoretical and methodological construction. 
Moreover, the Ashley et al. (2013) scale is derived from an existing media literacy scale (Primack et al. 2006), 
which allows comparison across multiple domains. Finally, the Maksl et al. (2015) scale focuses heavily on the 
cognitive underpinnings of media literacy - which we argue is only one important component of a news media 
literacy scale. 
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In addition, we extend the Ashley et al. (2013) model by adding two new components: measures for 
Self-Perceived Media Literacy (SPML) and Value for Media Literacy (VML). The SPML measures are 
designed to tap into individuals’ belief that they are in fact media literate and thus able to access, analyze and 
evaluate media content. Perceptions of one’s knowledge and actual knowledge are often related but distinct 
constructs - one can feel knowledgeable without actually holding correct beliefs (Hollander 1992; Radecki & 
Jaccard 1995). Moreover, SPML derives from Potter’s (2004) cognitive model of media literacy, which 
suggests that individuals need to feel in control of their media consumption and the media’s influence on them 
to become media literate. Similarly, Maksl et al. (2015) suggest that news media literacy scales should 
incorporate measures of mindful processing and belief in one’s ability to be in control of media messages, 
drawing heavily on Potter’s conceptualization of locus of control. They measure media locus of control and find 
that highly media literate teens feel more in control of their relationship with media. Therefore, we developed 
SPML to capture individuals’ perceptions about their abilities and confidence in their news media literacy skills. 
We expect that SPML will be a separate news media literacy dimension from those proposed by Ashley et al. 
(2013), but will be highly correlated with each of their subcomponents. 

Second, despite the interest in the relationship between news, news media literacy, and democracy, little 
research has focused specifically on measuring this aspect of news media literacy (Mihailidis 2009; 2014). 

News media literacy education often stresses the relationship between the news and citizens, highlighting the 
role of news as a key source of civic information (Burroughs, Brocato, Hopper, & Sanders 2009; Mihailidis 
2014). Therefore, we propose that news media literacy measures should also assess the value that individuals 
place on news media literacy for society. Value of Media Literacy (VML) should be a distinct concept from 
both the NML scale proposed by Ashley et al. (2013) and our proposed SPML scale. However, we also expect 
that the different components of news media literacy will remain highly interrelated. 

In sum, the inclusion of SPML and VML allows us to examine key components of news media literacy 
that are not measured in the original NML scale. Specifically, we consider individual perceptions about their 
news media literacy skills, tapping into notions of self-efficacy and competence that are critical to Potter’s 
(2004) cognitive model. VML allows us to measure the broader value that people place on news media literacy 
beyond their personal media use to begin to understand the relationship between news media literacy and 
society. Next I report on two studies conducted to validate the instruments with a larger sample of college 
undergraduates and a smaller sample of adults who completed the study online. 

Study 1: College Undergraduate Students 


Sample and Procedure 

Participants for this study were recruited from one of the two introductory communication courses at a 
large, diverse, public, Mid-Atlantic university (51% enrolled in Public Speaking, 49% enrolled in Interpersonal 
and Group Interaction). Both courses are small, face-to-face courses with fewer than 30 students per section. 
Because all students enrolled in the university must take one of these two courses to fulfill a general education 
requirement, typically during their first year, the students in these courses are a fairly representative cross- 
section of the lower-division undergraduate student body. 

All students in the courses were required to complete an online survey before the course began. The 
survey was available via Blackboard during the first two weeks of the semester. Students were given the option 
to opt in or out of having their results included in research studies, and all students who did not consent to 
having their work included in research were removed from the data set, per IRB instructions. Of the 1873 
students enrolled in the classes, 1481 completed the pre-course survey and agreed to allow their results included 
in research studies, for a response rate of 79%. Of the students who reported demographic data in the pre-course 
survey, 64.9% of participants were freshmen, 54.4% were female, 51.1% reported they were White or 
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Caucasian (with 22.8% Asian, 11.4% Black or African-American, and 8.8% Latino), with an average age of 
(M= 19.07, S.D. =2.87). 


Table 1 

Standardized Factor Loadings for News Media Literacy (AA, MM, RR), SPML and VML Scales 


Latent variable Item 

Standardized Factor 
Loading 

Study 1 

Study 2 


Authors & Audiences (AA) 

The owner of a media company influences the content that is 
produced 

0.716 

0.641 

News companies choose stories based on what will attract the 
biggest audience 

0.86 

0.744 

Individuals find news sources that reflect their own political 
values 

0.778 

0.723 

Messages & Meanings (MM) 

Two people might see the same news story and get different 
information from it 

0.764 

0.631 

People are influenced by news whether they realize it or not 

0.778 

0.628 

News coverage of a political candidate will influence people’s 
opinions 

0.789 

0.489 

News is designed to attract an audience’s attention 

0.818 

0.694 

Lighting is used to make certain people in the news look good or 
bad 

0.665 

0.485 

Production techniques can be used to influence a viewer’s 
perception 

0.819 

0.742 

When taking pictures, photographers decide what is most 
important 

0.699 

0.429 

Representations & Realities (RR) 

News makes things more dramatic than they really are 

0.68 

0.63 

A news story that has good pictures is more likely to show up in 
the news 

0.673 

0.569 

A news story about conflict is more likely to be featured 
prominently 

0.813 

0.809 

Self-Perceived Media Literacy (SPML) 

I have a good understanding of the concept of media literacy 

0.744 

0.733 

I have the skills to interpret news messages 

0.842 

0.773 

I understand how news is made in the U.S. 

0.752 

0.8 
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I am confident in my ability to judge the quality of news — 0.687 

I’m not sure what people mean by media literacy — -0.565 

I’m often confused about the quality of news and information — -0.386 


Value for Media Literacy (VML) 


Media literacy is important to democracy 

0.725 

0.675 

People should understand how media companies make decisions 
about news content 

0.859 

0.768 

People should accept information from the news on face value 

— 

-0.415 

It is the role of the press to represent diverse viewpoints 

0.646 

0.471 

The news media have a role to play in informing citizens about 
civic issues 

— 

0.722 

It is the job of citizens to overcome their own biases in consuming 
news 

0.733 

0.471 

People need to critically engage with news content 

— 

0.553 

The main purpose of the news should be to entertain viewers 

— 

-0.423 


Study 1 N = 1,481; Study 2 N = 330 

Results of Study 1 

We began by comparing two measurement models based upon the fifteen news media literacy items 
proposed by Ashley et al. (2013), one in which all of the items loaded onto a single latent variable, and a second 
in which the items loaded onto three separate latent variables representing the authors and audiences (AA), 
messages and meanings (MM), and representation and reality (RR) sub-dimensions proposed in the original 
research. Table 1 presents the items and factor loadings for Study 1 and Study 2. We used structural equation 
modeling in Mplus version 7 to conduct confirmatory factor analysis. We used the comparative fit index (CFI) 
and root mean square error of approximation (RMSEA) to evaluate model fit. According to Hu and Bentler 
(1999), a strong model fit is indicated by a CFI value of .95 or greater and a RMSEA value of .06 or less. To 
compare models, we used the chi-square statistic to assess which of the two rival models fit the data better 
(Holbert& Grill, 2015). 

Overall, the single latent variable model provided a worse fit for the data than the three latent variable 
model, as Table 2 shows. This worse fit for the single latent variable model appears for all measures of model 
fit examined: chi-square, CFI, and RMSEA. The statistical test confirms these observations: comparing the chi- 
square statistics for both models suggests that the three latent variable model is statistically superior than the 
single latent variable model (A%2[r//=3]= 1403.03 - 1142.42= 260.61, p<. 001). Therefore, we can conclude that 
a three latent variable model is preferable to a single latent variable model. 

However, although the model fit for the three latent variable model was comparatively better, it was still 
somewhat substandard in absolute terms. Therefore, we inspected the standardized model coefficients for fit 
with the model. Based on these observations, we made a number of modifications to the model. First, we 
dropped one item from the RR latent variable (“A journalist’s first obligation is to the truth.”) based on its poor 
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factor loading (.180) in comparison with the other factor loadings (ranging from .674 to .857). 1 Next, 
examination of the modification indices suggested the model fit could be improved by cross loading the MM 
item, “People pay more attention to news that fits with their beliefs than news that doesn’t,” onto the AA latent 
variable, which led us to believe this item was not an optimal measure of either construct, leading us to drop this 
item. 2 Finally, the modification indices indicated that two items “Lighting is used to make certain people in the 
news look good or bad” and “Production techniques can be used to influence a viewer’s perception” are 
functionally equivalent, artificially inflating their shared error variance. 3 Therefore, we covaried the error terms 
for these two items, which led to an adequate model fit (see Table 2). 

Table 2 

Model Fit Statistics for Study 1 



Single factor model 

Three factor model: 

AA, MM, RR 

Final three factor model: 
AA, MM, RR 

Final model: 

AA, MM, RR, SPML, 

VML 

r 

1403.03*** 

1142.42*** 

556.12** 

985.82*** 

df 

90 

87 

61 

159 

CFI 

.90 

.92 

.96 

.95 

RMSEA 

.099 [.095, .104] 

.091 [.086, .095] 

.074 [.069, .080] 

.059 [.056, .063] 


[90% confidence interval] 


Table 3 

Estimated Correlation Matrix For The Latent Variables 


Study 1 



AA 

MM 

RR 

SPML 

VML 

AA 

1.000 

- 

- 

- 

- 

MM 

0.907 

1.000 

- 

- 

- 

RR 

0.832 

0.880 

1.000 

- 

- 

SML 

0.532 

0.553 

0.533 

1.000 

- 

VML 

0.741 

0.787 

0.703 

0.707 

1.000 

Study 2 


AA 

MM 

RR 

SPML 

VML 

AA 

1.000 

- 

- 

- 

- 

MM 

0.981 

1.000 

- 

- 

- 

RR 

0.954 

0.929 

1.000 

- 

- 

SPML 

0.428 

0.445 

0.403 

1.000 

_ 


1 Model fit: yj (, df= 74)= 1094.23,p<.001, CFI= .923, RMSEA= .097, [.092, .102] 

2 Model fit: yj ( df= 62)= 762.8l,p<.001, CFI= .941, RMSEA= .087, [.082, .093] 

3 Following the logic of Holbert and Grill (2015), we believe this re-specification is justified because these two items are likely acting 
as functional equivalents that are measuring the same source of variance due to their similar conceptual focus. The high zero-order 
correlation between these items (r =.717, jt><.001) supports this argument. 

2 Model fit: yf ( df= 62)= 762.81, p<Ml, CFI= .941, RMSEA= .087, [.082, .093] 

3 Following the logic of Holbert and Grill (2015), we believe this re-specification is justified because these two items are likely acting 
as functional equivalents that are measuring the same source of variance due to their similar conceptual focus. The high zero-order 
correlation between these items (r =.717,/><.001) supports this argument. 
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VML _ 0.768 _ 0,735 _ 0.648 _ 0,546 _ 1.000 _ 

*A11 correlation coefficients significant at the jck.OOI level 

Using the modified three-latent variable model of NML established above, we next added the SML and 
VML items to the measurement model. This model fit the data well (see Table 2), and standardized factor 
loadings for each item in this model are reported in Table 1. As expected, both the SPML and VML variables 
were positively associated with each of the news media literacy subdimensions, as Table 3 shows, although it is 
worth noting that self-perceived media literacy (SPML) is the least correlated with the other news media 
literacy factors. Given this model fits the data well, we calculated the reliability coefficients and descriptive 
statistics for each of the latent variables uncovered by our Confirmatory Factor Analysis (see Table 4). These 
reliability coefficients indicated that the five separate subscales validated by the confirmatory factor analysis 
reliably measure their respective concepts. 

Table 4 

Descriptive Statistics for Proposed Measures 



Study 1 



Study 2 




Cronbach’s 

Alpha 

Mean 

Standard 

Deviation 

Cronbach’s 

alpha 

Mean 

Standard 

Deviation 

AA 

.83 

5.44 

1.07 

.74 

6.00 

.87 

MM 

.91 

5.53 

.94 

.80 

5.83 

.78 

RR 

.77 

5.28 

1.00 

.69 

5.85 

.92 

SPML 

.82 

4.87 

1.09 

.83 

5.11 

1.00 

VML 

.83 

5.35 

1.01 

.77 

5.64 

.81 

Study 1 N = 

1,481; Study 2 N = 

330 






Discussion 

Our analyses in Study 1 suggest that, contrary to the original scale proposed by Ashley et al. (2013), a 
three factor model separating out Authors and Audiences (AA), Messages and Meaning (MM), and 
Representation and Reality (RR) fits the data better than a single factor of News Media Literacy (NML). 
Further, our proposed new measures of Self-perceived Media Literacy (SML) and Value of Media Literacy 
(VML) are discrete constructs from the original items proposed by Ashley et al. (2013). This study confirms our 
initial proposition: that news media literacy is a multi-dimensional construct, requiring understanding of the 
diverse ways in which knowledge and attitudes can come together. Given that individuals’ perceptions of their 
own news media literacy and the value they place on news media literacy for democratic society are distinct 
from the news media literacy measures proposed by Ashley et al. (2013), we also anticipate that these 
constructs would differentially relate to news attitudes, knowledge, and habits. 

Study 2: Adults 

A second study was performed to test the content validity of these measures by comparing them to 
theoretically related constructs among a separate sample. We examined how well each of the five constructs 
revealed above relate to knowledge about news media structures and effects, knowledge of current events, and 
news media skepticism, as each of these have been proposed as an important correlate of news media literacy 
(Ashley et al. 2010, 2013; Maksl et al. 2015). We expected that some measures of news media literacy - and 
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especially how much people value news media literacy for democratic society - would prove more powerful in 
predicting attitudes and knowledge towards the media. 

To test these assumptions, we performed a survey using Amazon Mechanical Turk. Amazon Mechanical 
Turk is a crowd-sourcing service in which participants receive money for completing tasks. Research suggests 
that while the Mechanical Turk population is not representative of the US population, it is reliable and 
substantially more diverse than samples of US college students (Berinsky, Huber, & Lenz 2012; Casler, Bickel, 
& Hacked 2013). Therefore, this sample represents a distinct audience to test the measures of news media 
literacy scales, which has thus far only been tested with college students (Ashley et al. 2013). 

We recruited 330 participants during the summer of 2015 to participate in our survey. Participants were 
paid $0.60 after completing the survey. Our sample for this study is substantially older (M=35.73, SD= 12.60) 
and less racially diverse (83.6% white) than our previous sample, but similar in terms of gender (55.5% female). 
The sample remains relatively educated, with 53% having a 4-year college degree or higher. 

Variables Used in the Study 

News Media Knowledge: A total of 18 multiple-choice items were used to measure news media 
knowledge. These items were adapted from previous research (Ashley et al. 2013; Maksl et al. 2015) and 
included questions of media structure, ownership, content creation, and media effects. Each item was coded as 
correct or incorrect, then summed to create a news media knowledge score (M=13.47, SD= 3.58). 

Current Events Knowledge: Six multiple-choice items asked people about current events, such as which 
party controls the US Senate, the US unemployment rate, and the number of female justices on the Supreme 
Court, adapted from other scales (Maksl et al. 2015; Pew, 2015). Each question was scored as correct or 
incorrect and summed to create a current events knowledge score (M=4.39, SD= 1.20). 

News Media Skepticism: Participants rated their agreement on seven-point scales for four items to 
measure media skepticism, including whether the news media is trustworthy, accurate, gets in the way of 
society solving its problems, and confidence in the press (Maksl et al. 2015). These items were averaged to 
create an index (a=.78, M=4.70, SZ)=1.11). 

News Media Literacy Measures'. Our measures for Authors and Audiences (AA), Messages and 
Meaning (MM), and Representation and Reality (RR) were identical to Study 1. However, additional items 
were included to measure Self-perceived Media Literacy (SPML) and Value of Media Literacy (VML). These 
items were added to further develop these constructs, which we discuss in more detail in the results section. 

Control Variables 

Need for Cognition: Four items were used to measure need for cognition, which ranged on a seven-point 
scale from strongly disagree to strongly agree (Maksl et al., 2015). These items were averaged to create an 
index (a=.83, M=4.95, SD=133). 

Locus of Control: Six items were used to measure media locus of control, which measures how much 
control individuals believe they have over media effects (Maksl et al., 2015). These items were averaged to 
create an index (a=.63, M=4.85, SD=.80). 

Other Control Variables: In addition, our models control for demographic variables, including age, 
gender, education, race, and ethnicity (4.3% Hispanic or Latino), as well as political orientations. Participants 
were asked their political ideology in terms of social issues and behavior, economic issues, and national security 
issues ranging from very liberal to very conservative, which was averaged to form an index (a=.90, M= 3.63, 
SD= 1.66). Participants also answered their party identification on a seven-point scale ranging from Strong 
Republican to Strong Democrat (M=4.49, SD= 1.70), and this scale was folded to create a measure of partisan 
strength ranging from 0 to 3 (M=1.49, SD=. 95). Finally, people were asked how important politics was to them 
personally on a seven-point scale (M=4.26, SD= 1.72) as a measure of political interest. 
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Study 2 Results 

Confirmatory Factor Analysis. First, we replicated the previous confirmatory factor analysis (CFA) for 
the five media literacy scales identified in Study 1. Previous research to date has only examined these scales 
among undergraduate populations, which might produce different factor structures than a more diverse sample, 
especially one less embedded in the education system. As in Study 1, we use a CFI value of .95 or greater and a 
RMSEA value of .06 or less to indicate strong model fit (Hu & Bentler, 1999). These analyses confirm that a 
precise replication of the model from Study 1 produces a strong model fit: the CFI equals .95 and the RMSEA 
is below .05 (see Table 5). These findings validate our findings from Study 1 and suggest that these measures 
are appropriate beyond a student sample. 

However, all five scales proposed in Study 1 suffered from a methodological flaw. Previously, all of the 
items in the study were coded in the same direction - meaning that a higher score always indicates higher media 
literacy. As such, these items are likely to suffer from response biases, including acquiescence bias (individuals 
are more likely to agree with survey items) and careless response (Frey, Botan, & Kreps, 2000; Weijters, 
Baumgartner, & Schillewaert, 2013). In the second study, we proposed using additional reverse-coded measures 
for SPML and VML to limit response biases. 

The confirmatory factor analysis for the revised model, including the new items for SPML and VML, 
suggests model fit is mediocre (see Table 5). However, while adding reverse-coded items is methodologically 
sound, previous research suggests they often hinder model fit (Weijters et ah, 2013). Therefore, we correlate the 
errors for the reverse-coded items in the SPML and VML scales, which are likely to have shared error due to 
the reverse-coded nature of the items. This improves model fit, but it remains substandard. 4 Finally, as in Study 
1, we examine the model modification indices to determine if additional changes to the model are required. A 
single modification stood out: correlating the errors for two measures of SML: “I’m not sure what people mean 
by media literacy” and “I have a good understanding of the concept of media literacy.” Given the similarity in 
question construction, we feel comfortable correlating the error variance of these items. Altogether, these 
changes produce an adequate fitting model: the CFI is lower than is ideal at .912, but the RMSEA remains good 
at .054. The final factor loadings for each measure, the correlation among the latent variables, and the 
descriptive statistics for each scale are available in to facilitate comparison to the Study 1 results. 5 

Table 5 

Model Fit Statistics for Study 2 



Replicated model 

Initial extended model 

Final extended model 

r 

286.43*** 

629.34*** 

607.60*** 

df 

159 

313 

310 

CFI 

.95 

.89 

.91 

RMSEA 

.049 [.040, .058] 

.061 [.055, .067] 

.054 [.048, .060] 


[90% confidence interval] 


Predictive and Construct Validity. Having replicated our five factor model using confirmatory factor 
analysis, we move to testing the predictive validity of the items - namely, their ability to tap into related 


4 Model fit: f ( df= 311)= 654.12,p<.001, CFI= .90, RMSEA= .058, [.052, .064] 

5 Though the absolute value for some factor loadings for study 2 are below .5, it is recommended that factor loadings with an absolute 
value greater than .4 should be interpreted, and some scholars use the cutoff of .3 (Field, 2013, p. 681L CIT. Furthermore, the 
reliability coefficients for each of the subscales in Study 2 are at acceptable levels, and the addition of reverse-coded items to the 
VML and SPML scales adds internal validity by addressing response style effects (Frey, Botan, & Kreps, 2000, p. 102). For these 
reasons, we recommend using all of the items from study 2 to reconstruct the VML and SPML scales in future research. 
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theoretical constructs - using a series of regression analyses. Overall, these regression analyses broadly support 
our expectations about the differential impact of diverse components of news media literacy on knowledge and 
attitudes towards the media, as Table 6 shows. 

Across these models, two components of news media literacy stand out in predicting news media 
knowledge, current events knowledge, and news media skepticism. First, individuals who more strongly value 
media literacy (VML) are more likely to have higher levels of news media knowledge (b=.35,/?<.001), more 
knowledge of current events (b=.19,/?<.05), and greater skepticism towards the news media (b=. 13, p<. 10). 
Further, for both types of knowledge, VML is the most consistent and one of the strongest predictors of 
knowledge, above and beyond demographic and political factors. Second, higher levels of reported 
understanding of authors and audiences (AA) is positively related to both news media knowledge (b=. 18, 
7?<.01) and news media skepticism (b=.28,/?<.01), but not to current events knowledge. The other types of 
news media literacy - knowledge of messages and meaning (MM), representation and reality (RR), or stronger 
self-perceived media literacy (SPML) - produce no consistent effects on these variables. 

Table 6 

Regression Analyses for Study 2 



News Media Knowledge 

Current Events 
Knowledge 

Media Skepticism 

Age 

.06 

20 *** 

-.07 

Gender 

-.13** 

-.18** 

-.03 

Education 

2i*** 

.09 

-. 12 * 

Race 

. 11 * 

. 11 * 

-.01 

Ethnicity 

j 

-.02 

.06 

Adjusted R-squared 

.777*** 

777 *** 

.003 

Political ideology (conservative high) 

-.18** 

-.17* 

-.05 

Party identification (Democrat high) 

-.02 

-.06 

-,15 + 

Partisan strength 

-.09* 

-.00 

-.06 

Political interest 

-.07 

.05 

.01 

Adjusted R-Squared 

.255*** 

.148** 

.009 

Need for cognition 

.02 

-.02 

-.04 

Locus of control 

-.05 

-.05 

-. 12 * 

Adjusted R-squared 

2#7*** 

.7 46 

.011 

AA 

.18** 

.04 

.28** 

MM 

-.01 

.04 

-.09 

RR 

, 11 + 

.02 

.13 

SPML 

-.02 

-.00 

.04 

VML 

35*** 

.19* 

.13 + 

Adjusted R-squared 

.570*** 

.180** 

743 *** 


Standardized beta coefficients are reported to facilitate comparison. 
><.10, ><.05, *><.01, **><.001 


Discussion 

The results from these two studies shed light on the best approaches to measure the multi-dimensional 
concept of news media literacy. This study confirms the value of conceptualizing news media literacy using the 
theoretical subcomponents that the Ashley and colleagues (2013) used in their study. But additional measures 
are needed to capture the diverse nature of news media literacy: self-perceived media literacy (SPML) and value 
of media literacy for democratic society (VML), and in this study we confirmed that these variables function 
separately from the measures proposed by Ashley et al. (2013). 
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Study 2 allowed us to validate these proposed relationships with a more diverse sample of American 
adults. To our knowledge, this study is the first to use measures of news media literacy with a non-college 
student sample. Further, Study 2 demonstrates that these distinct measures of news media literacy differentially 
relate to measures of knowledge - both of media structures and effects and of current events - and skepticism 
towards the news media. In fact, our new proposed measure on the value of media literacy (VML) is the most 
consistent predictor of higher levels of knowledge, both of the media itself and of current events, and more 
skepticism towards the press. Meanwhile, knowledge of authors and audiences (AA) stands out among the 
previous news media literacy measures in predicting knowledge and skepticism towards the media. The other 
components of news media literacy proposed in the original NML scale - attitudes about messages and meaning 
(MM) or representation and reality (RR) - prove less consequential in predicting these outcomes, despite the 
high correlations among these concepts. Therefore, Study 2 provides strong support for our argument that these 
represent in fact distinct forms of news media literacy, even if they are closely related. However, future research 
is needed to examine the mechanisms behind these results - and especially, what makes value of media literacy 
and knowledge of authors and audiences particularly consequential for attitudes towards media content. 

Interestingly, self-perceived media literacy (SPML) never is significantly related to any of our 
dependent variables. This may suggest that people’s feelings of their own news media literacy are untethered 
from their actual experiences and knowledge of the news media, similar to other domains such as health and 
political information (Hollander 1992; Radecki & Jaccard 1995). Other researchers have found that self- 
reported measures of perceived learning produce only a weak relationship with performative measures of 
learning (Hooker & Denker 2013), so it is possible that individuals are similarly inaccurate when estimating 
their own levels of news media literacy. The lack of relationships between SPML and our outcomes is further 
reinforced by the null findings of the effects of media locus of control, in contrast to previous research (Maksl et 
al. 2015; Potter 2004). Future research should consider when individuals’ beliefs about their media literacy are 
most likely to be powerful, or whether their effects are indirectly funneled through more precise measures of 
understanding of media systems (such as AA) or value for media literacy (VML). 

Two alternative explanations may explain the limited effects of self-perceived media literacy in this 
study. First, it may be that self-perceived media literacy is consequential in promoting other democratically 
valuable orientations, such as civic engagement or exposure to diverse views (Kahne et al. 2012; Mihailidis 
2009; 2014). Second, self-perceived media literacy may shape information-seeking behavior intended to 
improve one’s media literacy in competing ways depending on individual characteristics. For some, relatively 
high levels of self-perceived media literacy may lead to the conclusion that one is sufficiently media literate and 
therefore no further development of the skill is needed. Alternatively, self-perceived media literacy may 
increase information-seeking behavior if individuals view the continuous development and maintenance of their 
media literacy knowledge as important to their self-concept as a media literate citizen. Future research should 
extend these findings to examine how diverse beliefs and attitudes towards news media literacy predict a range 
of democratic outcomes. 

Of course, this study has limitations that need to be acknowledged. While we use two separate samples 
to test the structure of news media literacy measures, neither sample is representative of the U.S. population. It 
may be that the components of news media literacy differ among a less educated sample, which future research 
should test. Additionally, while our analyses consistently demonstrate that the five components of news media 
literacy proposed here are in fact distinct constructs, they remain highly correlated, and we cannot test whether 
a single higher-order measure of news media literacy also drives all of the subcomponents examined here. It 
may also be that in some cases, a more parsimonious approach to news media literacy may be appropriate, in 
which Ashley et al.’s (2013) three scales could be combined into a single measure of news media literacy, as 
Ashley and colleagues initially propose. 

We urge scholars interested in news media literacy to carefully consider the multi-dimensional nature of 
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this concept in their measurements and study design. Our findings suggest that research into news media 
literacy needs to go beyond measuring simply knowledge of media systems and effects to also consider both 
self-perceived media literacy and value that individuals have for news media literacy. Moreover, relevant news 
media literacy attitudes may differ depending on the outcomes being considered; for example, in some domains 
we might expect self-perceived news media literacy to be most influential, whereas in other situations 
understanding of authors and audiences or higher value for news media literacy may be more powerful. Given 
the importance of news media literacy in navigating increasingly complicated news environments, appropriate 
care must be taken to capture the complex nature of the construct in order to perceive effects and motivate 
educational efforts. 

References 

Ashley, S., Poepsel, M., & Willis, E. 2010. “Media Literacy and News Credibility: Does Knowledge of Media 
Ownership Increase Skepticism in News Consumers?” Journal of Media Literacy Education 2: 37-46. 
Ashley, S., Maksl, A., & Craft, S. (2013). “Developing a News Media Literacy Scale.” Journalism and Mass 
Communication Educator, 68: 7-21. 

Berinsky, A. J., Huber, G. A., & Lenz, G. S. 2012. “Evaluating Online Labor Markets for Experimental 
Research: Amazon.com’s Mechanical Turk.” Political Analysis, 20: 351-368. 

Burroughs, S., Brocato, K., Hopper, P. L., & Sanders, A. 2009. “Media Literacy: A Central Component of 
Democratic Citizenship.” The Educational Forum, 73: 154-167. 

Casler, K., Bickel, L., & Hackett, E. 2013. “Separate but Equal? A Comparison of Participants and Data 
Gathered via Amazon’s MTurk, Social Media, and Face-to-face Behavioral Testing.” Computers in 
Human Behavior, 29(6): 2156-2160. 

Chu, D., & Lee, A. Y. L. 2014. “Media Education Initiatives by Media Organizations: The Uses of Media 
Literacy in Hong Kong Media.” Journalism and Mass Communication Educator, 69: 127-145. 

Duran, R. L., Yousman, B., Walsh, K. M., & Longshore, M. A. 2008. “Holistic Media Education: An 
Assessment of a College Course in Media Literacy.” Communication Quarterly, 56: 49-68. 

Fleming, J. 2013. “Media Literacy, News Literacy, or News Appreciation? A Case Study of the News Literacy 
Program at Stony Brook University.” Journalism and Mass Communication Educator, 69(2): 146-165. 
Field, A. 2013. Discovering Statistics using SPSS, 4 th edition. Thousand Oaks CA: Sage. 

Frey, L. R., Botan, C. H., & Kreps, G. L. 2000. Investigating Communication (2 nd ed.). Boston: Allyn and 
Bacon. 

Hobbs, R. 2010a. Digital and Media Literacy: A Plan of Action. Washington, DC: The Aspen Institute. 

Hobbs, R. 2010b. “News Literacy: What Works and What Doesn’t.” Paper presented at the Association for 
Education in Journalism and Mass Communication Conference, Denver, Colorado. 

Hobbs, R. 2011. “The State of Media Literacy: A Response to Potter.” Journal of Broadcasting and Electronic 
Media, 55: 419-430. 

Hobbs, R., & Frost, R. 2003. “Measuring the Acquisition of Media-literacy Skills.” Reading Research 
Quarterly 38: 330-355. 

Holbert, R. L., & Grill, C. 2015. “Clarifying and Expanding the use of Confirmatory Factor Analysis in 
Journalism and Mass Communication Research.” Journalism and Mass Communication Quarterly. 
Advance online publication. 

Hollander, B. A. (1995). “The New News and the 1992 Presidential Campaign: Perceived vs. Actual Political 
Knowledge. Journalism and Mass Communication Quarterly, 72: 786-798. 

Hooker, J. & Denker, K. 2013. “The Learning Loss Scale as an Assessment Tool: An Empirical Examination of 
Convergent Validity with Performative Measures. Communication Teacher, 28: 130-143. 


52 



E. Vraga, M. Tully, J. Kotcher, A. Smithson & M. Broeckelman-Post / Journal of Media Literacy Education 7(3), 41 - 53 


Hu, L., & Bentler, P. M. 1999. “Cutoff Criteria for Fit Indices in Covariance Structure Analysis: Conventional 
Criteria Versus New Alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6: 1-55. 

Kahne, J., Lee, N., & Feezell, J. T. 2012. “Digital Media Literacy Education and Online Civic and Political 
Participation.” International Journal of Communication, 6: 1-24. 

Livingstone, S. 2004. “Media Literacy and the Challenge of New Infonnation and Communication 
Technologies.” The Communication Review, 7: 3-14. 

Maksl, A., Ashley, S., & Craft, S. 2015. “Measuring News Media Literacy.” Journal of Media Literacy 
Education, 6(3): 29-45. 

Mihailidis, P. 2009. “The First Step is the Hardest: Finding Connections in Media Literacy Education.” Journal 
of Media Literacy Education, 1(1): 53-67. 

Mihailidis, P. 2014. Media Literacy and the Emerging Citizen: Youth, Engagement and Participation in Digital 
Culture. New York: Peter Lang. 

Mitchell, A., & Page, D. 2015, June 1. “Millennials and Political News: Social Media - The Local TV for the 
Next Generation.” Pew Research Center. Retrieved from: 
http://www.ioumalism.org/fdes/2015/06/Millennials-and-News-FINAL.pdf 

Pew Research Center. 2015, April 28. The News IQ Quiz. Pew Research Center. Retrieved from: 

http://www.people-press.org/2015/04/28/what-the-public-knows-in-pictures-words-maps-and-graphs/ 

Potter, J. W. 2004. Theory of media literacy: A cognitive approach. Thousand Oaks, CA: Sage. 

Potter, W. J. 2010. “The State of Media Literacy.” Journal of Broadcasting and Electronic Media, 54: 675-696. 

Primack, B. A., Gold, M. A., Switzer, G. E., Hobbs, R., Land, S. R., & Fine, M. J. 2006. “Development and 
Validation of a Smoking Media Literacy Scale for Adolescents.” Archives of Pediatrics and Adolescent 
Medicine, 160(4): 369-374. 

Radecki, C. M., & Jaccard, J. 1995. “Perceptions of Knowledge, Actual Knowledge, and Information Search 
Behavior.” Journal of Experimental Social Psychology, 31(2): 107-138. 

Shearer, E. 2015, July 14. “Five Key Takeaways about Twitter, Facebook, and News Use.” Pew Research 
Center. Retrieved from: http://www.pewresearch.org/fact-tank/2015/07/14/5-kev-takeaways-about- 
twitter-facebook-and-news-use/ 

Tully, M., & Vraga, E. K. Forthcoming. “Effectiveness of a Media Literacy Advertisement in Partisan Versus 
Monpartisan Online Media Contexts.” Journal of Broadcasting & Electronic Media. 

Vraga, E. K., Tully, M., & Rojas, H. 2009. “Media Literacy Training Reduces Perceptions of Bias.” Newspaper 
Research Journal, 30: 68-81. 

Vraga, E. K., & Tully, M. 2015. “Media Literacy Messages and Hostile Media Perceptions: Processing of 
Nonpartisan Versus Partisan Political Information.” Mass Communication and Society, 14, 422-448. 

Weijters, B., Baumgartner, H., & Schillewaert, N. 2013. “Reversed Item Bias: An Integrative Model. 
Psychological Methods, 18: 320-334. 


53 







